System and methods for medical image quality assessment using deep neural networks

ABSTRACT

The current disclosure provides methods and systems for rapidly and consistently determining medical image quality metrics following acquisition of a diagnostic medical image. In one embodiment, the current disclosure teaches a method for determining an image quality metric by, acquiring a medical image of an anatomical region, mapping the medical image to a positional attribute of an anatomical feature using a trained deep neural network, determining an image quality metric based on the positional attribute of the anatomical feature, determining if the image quality metric satisfies an image quality criterion, and displaying the medical image, the image quality metric, and a status of the image quality criterion via a display device. In this way, a diagnostic scanning procedure may be expedited by providing technicians with real-time insight into quantitative image quality metrics.

FIELD

Embodiments of the subject matter disclosed herein relate to medicaldiagnostic imaging. In particular, systems and methods are provided forautomatic evaluation of medical image quality using deep neuralnetworks.

BACKGROUND

Image quality assessment is routinely performed by a technician or otherpersonnel following acquisition of a medical image, and may inform thedecision to proceed with the currently acquired medical image, or rejectthe medical image and re-scan the imaging subject. The technician mayevaluate the medical image based on various technical factors dependingon the type of diagnostic imaging being performed. In one example,during acquisition of a chest x-ray, a technician may evaluate the x-rayimage based on lung coverage, a degree of patient rotation, timing ofthe image acquisition relative to the inspiration/expiration cycle,x-ray beam penetration, etc. Further, a radiologist may evaluate similartechnical factors when making a diagnosis based on a medical image.Visually inspecting a medical image to determine if the image satisfiesthe various relevant image quality criteria may reduce the speed of thescanning process, as an imaging technician may need to make such anevaluation in order to determine if a scanning procedure should berepeated. Further, visually assessing image quality may introduce anelement of subjectivity and variability to the diagnostic imagingprocess, as there may be variation between the assessments of differenttechnicians/radiologists on a single image, or between different imagesassessed by a single technician/radiologist. In particular, humanassessment of quantitative geometric values in medical images may beprone to variation and may lack precision. Thus, it is generally desiredto provide systems and methods for automated image quality assessment,particularly in assessment of quantitative image quality metrics.

SUMMARY

The present disclosure teaches systems and methods which at leastpartially address the issues described above. In one embodiment, animage quality assessment may be automated by a method comprising,acquiring a medical image of an anatomical region, mapping the medicalimage to a positional attribute of an anatomical feature using a traineddeep neural network, determining an image quality metric based on thepositional attribute of the anatomical feature, determining if the imagequality metric satisfies an image quality criterion, and responding tothe image quality metric not satisfying the image quality criterion bydisplaying the medical image, the image quality metric, and anindication of the unsatisfied image quality criterion, via a displaydevice.

By automatically determining if a medical image satisfies relevant imagequality criteria of a particular imaging protocol, a scanning proceduremay be expedited compared to conventional approaches. Further, bydisplaying the medical image along with the image quality metric,wherein the image quality metric may provide quantitative informationregarding one or more aspects of a medical image, more rapid, precise,and repeatable assessment and comparison of medical images may befacilitated.

The above advantages and other advantages, and features of the presentdescription will be readily apparent from the following DetailedDescription when taken alone or in connection with the accompanyingdrawings. It should be understood that the summary above is provided tointroduce in simplified form a selection of concepts that are furtherdescribed in the detailed description. It is not meant to identify keyor essential features of the claimed subject matter, the scope of whichis defined uniquely by the claims that follow the detailed description.Furthermore, the claimed subject matter is not limited toimplementations that solve any disadvantages noted above or in any partof this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

The present disclosure will be better understood from reading thefollowing description of non-limiting embodiments, with reference to theattached drawings, wherein below:

FIG. 1 shows an embodiment of a method for automatically assessing imagequality criteria using a deep neural network;

FIG. 2 shows an embodiment of a medical imaging system configured toautomatically assess image quality criteria using deep neural networks;

FIG. 3 shows an embodiment of a method for determining a first imagequality metric which may be used to assess an image quality criterion;

FIG. 4 shows an embodiment of a method for determining a second imagequality metric which may be used to assess an image quality criterion;

FIG. 5 shows an embodiment of a method for determining a third imagequality metric which may be used to assess an image quality criterion;

FIG. 6 shows an embodiment of a graphical user interface for displayingmedical images and the first image quality metric;

FIG. 7 embodiments of a graphical user interface for displaying medicalimages and the second image quality metric;

FIGS. 8A and 8B show an embodiments of a graphical interface fordisplaying medical images and the third image quality metric;

FIG. 9 shows a first embodiment of a deep neural network configured tomap medical images to a positional attribute an anatomical feature; and

FIG. 10 shows an embodiment of a multi-tasking deep neural networkconfigured to map medical images to two or more positional attributes.

DETAILED DESCRIPTION

The following description provides systems and methods for automaticdetermination of image quality metrics using deep neural networks,wherein said image quality metrics may be used in the evaluation of oneor more image quality criteria. Image quality metrics may comprisequantitative geometrical aspects of a medical image, such as distancebetween anatomical features (e.g., a mediastinal width), angles formedby anatomical features (e.g., a costophrenic angle), position of animaging subject relative to an imaging device (e.g., a degree ofrotation of an imaging subject with respect to a projection plan of andiagnostic scan), relative sizes of anatomical features (e.g., acardiothoracic ratio), as well as other aspects which may be indirectlyderived from geometrical aspects of a medical image, such as an extentof inspiration/expiration of the lungs of an imaging subject or anextent of overlap between anatomical features. Such geometrical aspectsmay be difficult for a human, such as an imaging technician orradiologist, to quantify with both precision and consistency. Further,different radiologists/technicians may arrive at different estimationsfor such geometrical aspects, which may introduce an element ofvariability into imaging/diagnostic workflows. In particular, duringmedical image acquisition, determinations of image quality, which mayinform a technician's decision whether to accept an acquired medicalimage or repeat a scan, may be based on said geometrical aspects. Humanestimation of geometrical aspects may, in some cases, be a timebottleneck in acquisition of medical images. Thus, it is generallydesired to provide systems and methods to quickly, consistently, andprecisely, determine such quantitative image quality metrics,particularly in the context of medical image acquisition.

The inventors herein have at least partially addressed the above issues,by providing systems and methods for automatically determining saidimage quality metrics, using a combination of machine-learning basedinference of one or more positional attributes of one or more anatomicalfeatures, and expert logic based determination of clinically relevantimage quality metrics from said positional attributes. Further, theinventors herein disclose systems and methods for integrating saidapproach into rapid and computationally efficient determination of imagequality criteria during a medical imaging procedure, using the imagequality metrics so determined.

In one embodiment, a medical imaging system 200, shown in FIG. 2 , mayautomatically determine one or more image quality metrics, and use saidimage quality metrics to evaluate image quality criteria pertaining to aparticular imaging protocol, by implementing method 100 shown in FIG. 1. Image quality metrics determined by method 100 may include, but arenot limited to, a first image quality metric, a second image qualitymetric, and a third image quality metric, the determination of which isdescribed in the discussion of method 300, shown in FIG. 3 , method 400,shown in FIG. 4 , and method 500, shown in FIG. 5 , respectively. Thefirst image quality metric, the second image quality metric, and thethird image quality metric, may be graphically presented to atechnician/radiologist or other user, via the graphical user interface(GUI) 600 shown in FIG. 6 , the GUI 700, shown in FIG. 7 , and the GUI800, shown in FIG. 8 , respectively. Exemplary deep neural networkarchitecture 900 and multi-tasking neural network 1000, shown in FIGS. 9and 10 , respectively, may be used by medical imaging system 200, toinfer positional attributes of one or more anatomical features, such asduring execution of method 100.

Turning to FIG. 1 , an embodiment is shown of a method 100 forautomatically determining one or more image quality metrics, andassessing image quality criteria based on the image quality metrics.Method 100 may be executed by an imaging system, such as imaging system200 discussed below with reference to FIG. 2 .

At operation 102, the medical imaging system receives an imagingprotocol selection, wherein the imaging protocol selection uniquelyidentifies a type of diagnostic medical image to be acquired. In someembodiments, the medical imaging system may automatically suggest animaging protocol based on one or more features, such as camera imagesobtained of an imaging subject prior to diagnostic imaging. Thesuggested imaging protocol may be approved by a user, and said approvalmay constitute an imaging protocol selection. In some embodiments, theimaging protocol selection may indicate one or more acquisitionparameters to be applied during acquisition of a medical image. In someembodiments, the imaging protocol selection includes, or is linked with,one or more imaging quality criteria, as well as locations in memory ofone or more deep neural networks and instructions for determining animage quality metric from one or more positional attributes inferred bysaid one or more deep neural networks. The imaging protocol selectionmay be input by a user, such as a technician or radiologist, via a userinput device of the medical imaging system.

At operation 104, the medical imaging system pre-loads the deep neuralnetwork(s) and image quality criteria included or linked with theimaging protocol selection. In some embodiments, the imaging protocolselection includes locations in memory of one or more deep neuralnetworks associated with the imaging protocol selection, and atoperation 104 the medical imaging system may retrieve and load intoactive memory, said one or more deep neural networks from the locationsin memory. By pre-loading the deep neural networks into active memory ofthe medical imaging system, prior to acquisition of a medical image (atoperation 106), a latency of the automatic determination of the imagequality metrics may be reduced. Commonly radiologists may want animaging subject positioned consistently for subsequent imaging, so theradiologist can assess variations caused by diseaseprogression/improvement, and not changes caused by variation in imagequality or imaging subject position. Therefore, in some embodiments, ifan imaging subject has previously been imaged using a particular imagingprotocol, the imaging protocol selection may include a uniquepatient/imaging subject identifier, such as an alpha numeric ID number.In such embodiments, the image quality criteria may be based on priorimages acquired for the imaging subject/patient indicted by theidentifier. In one example, assuming a prior image acquired using aparticular imaging protocol for an imaging subject satisfied the imagequality criteria associated with the imaging protocol, at operation 104the medical imaging system may load a patient specific image qualitycriteria based on the prior image. In one example, the image qualitycriteria may comprise a match score threshold, wherein a currentlyacquired medical image may dissatisfy the image quality criteria if amatch score, determined based on an extent of deviation between thecurrently acquired image and the previously acquired image, is less thanthe match score threshold.

At operation 106, the medical imaging system acquires a medical image ofan anatomical region using a medical imaging device. In someembodiments, operation 106 includes the medical imaging device settingone or more acquisition parameters of the imaging device based on theimaging protocol selection. Acquisition parameters may includeorientation of a radiation source, dose timing/amount, receiver gain,and other medical image acquisition settings known in the art. Themedical imaging system may apply the acquisition parameters, scan ananatomical region of an imaging subject/patient to acquire imaging data,and perform various image reconstruction procedures on the imaging datato produce a medical image. The image acquired at operation 106 may be atwo-dimensional (2D) or three-dimensional (3D) image. The imagingmodality used to acquire the medical image at operation 106 maycomprise, but is not limited to, x-ray imaging, computed tomography(CT), magnetic resonance imaging (MRI), and positron emission tomography(PET).

At operation 108, the medical imaging system maps the medical image toone or more positional attributes of one or more anatomical featuresusing a trained deep neural network. The medical imaging system may feedthe medical image acquired at operation 106 into an input layer of theone or more deep neural networks loaded into active memory at operation104, wherein the deep neural network(s) may extract and/or encodefeatures from the medical image, and map said features to one or morepositional attributes of one or more anatomical features, as discussedin more detail in the description of FIGS. 9 and 10 . In someembodiments, positional attributes include one or more of a location ofan anatomical feature in the medical image, a segmentation mask of oneor more anatomical features, an area or volume of the medical imageoccupied by an anatomical feature, an orientation of an anatomicalfeature (e.g., a direction of extent of one or more pre-determined axesof an anatomical feature relative to a coordinate system of the medicalimage), a length of an anatomical feature, a width of an anatomicalfeature, a breadth of an anatomical feature, and a classification scoreof an anatomical feature, wherein the classification score indicates aprobability or binary (yes or no) assessment of if an anatomical featureis present in a field of view of the medical image.

At operation 110, the medical imaging system determines one or moreimage quality metrics based on the positional attributes determined atoperation 108. Methods 300, 400, and 500, discussed below with referenceto FIGS. 3, 4, and 5 , respectively, discuss in more detail threeexemplary embodiments of three distinct classes of image qualitymetrics, and methods for determining said image quality metrics frompositional attributes. It will be appreciated that the image qualitymetrics, and associated methods of determination, discussed in FIGS. 3,4, and 5 , are given as exemplary embodiments, and are not to beconstrued as limiting the current disclosure. In some embodiments, imagequality metrics include, a costophrenic angle, a cardiothoracic ratio, amediastinal width, an angle of rotation of an imaging subject withrespect to a projection plane of the medical image, an angle of rotationof an imaging subject with respect to a plane perpendicular to theprojection plane of a medical image, and an extent of inspiration oflungs of the imaging subject.

At operation 112, the medical imaging system determines if the one ormore image quality metrics satisfy the corresponding image qualitycriteria. An image quality criterion may comprise a value, or range ofvalues, wherein an image quality metric is said to satisfy thecorresponding image quality criterion if the image quality metric equalsthe value, or falls within the range of values, indicated by the imagequality criterion. Likewise, an image quality metric not equaling thevalue, or not falling within the range of values, is said to not satisfy(or dissatisfy) the image quality criterion. As discussed above withreference to operation 104, the medical imaging system may retrieve theimage quality criteria based on the imaging protocol selection. In someembodiments, the imaging protocol selection links or points to, alocation in memory of the medical imaging system where the image qualitycriteria associated with the imaging protocol selection are stored. Insome embodiments, image quality criteria of distinct imaging protocolsmay be associated with different values, or ranges of values, even whensaid value or ranges of values are associated with a same image qualitymetric. As an example, a first imaging protocol, indicated by an imagingprotocol selection received by the medical imaging system, may indicatea first pre-determined range of rotation angles for an imaging subject,whereas a second imaging protocol may indicate a second, distinct (thatis, non-equivalent) range of rotation angles.

At operation 114, the medical imaging system displays the medical image,the image quality metrics, and the image quality criteria via a displaydevice. Exemplary embodiments of GUIs which may be displayed atoperation 114 are shown in FIGS. 6, 7A, 7B, and 8 . In some embodiments,the medical imaging system may highlight the one or more anatomicalfeatures of the medical image for which positional attributes weredetermined at operation 108. In some embodiments, the medical imagingsystem may display the medical image, and the image quality metric, butnot the image quality criteria. In some embodiments, the medical imagingsystem may display a status of the image quality criteria, wherein thestatus indicates if the currently displayed medical image satisfies theimage quality criteria, based on the image quality metric. In someembodiments, the medical imaging system may display an image qualitymetric proximal to the one or more anatomical features used in thedetermination of said image quality metric. In embodiments where theimage quality metric is an angle formed by one or more anatomicalfeatures, the medical imaging system may display a numerical value ofthe angle, as well as a visual indication of the angle, at a locationcorresponding to said angle. The medical imaging system may display themedical image along with one or more of the image quality metric, andstatus of one or more image quality criteria, in substantially real timefollowing image acquisition at operation 106, which may be facilitatedby operation 104, wherein the relevant deep neural network(s) and imagequality metrics are pre-loaded, prior to acquisition of the medicalimage.

At operation 116, the medical imaging system stores the image qualitymetric as metadata of the medical image. In some embodiments, the imagequality metric, along with the statuses of the one or more image qualitycriteria, may be stored as metadata, or otherwise linked, to the medicalimage. In some embodiments, the image quality metric may be stored inthe DICOM header of a medical image. In this way, later analysis of themedical image, such as by a radiologist, may be further facilitated byease of access to image quality metric data.

At operation 118, the medical imaging system adjusts the medical imagingdevice based on the satisfied/dissatisfied image quality criteria. Inone example, the medical imaging system may automatically reposition theimaging device relative to an imaging subject based on the image qualitycriteria, such as by moving the imaging device up, down, left, right, oraltering an angle with respect to the imaging subject, to account forpositioning errors of the imaging subject indicated by the image qualitycriteria. In another example, the medical imaging system may adjust oneor more imaging parameters, such as an intensity of radiation usedduring image acquisition, a gain of a receiver, or other imagingparameters which may compensate for a dissatisfied image qualitycriteria. In one example, at operation 118, the medical imaging systemmay display a suggestion to a user via a display device to reposition animaging subject based upon the satisfied/dissatisfied image qualitycriteria. In one example, the medical imaging system may display asuggestion for an imaging subject to change a degree of rotation withrespect to an imaging device. In some examples, at operation 118, inresponse to a dissatisfied image quality criteria indicating a field ofview is too large, the medical imaging system may adjust the imagingdevice by decreasing the collimation area by actuating collimationblades of the imaging device. In another example, in response to adissatisfied image quality criteria indicating a field of view is toosmall, the medical imaging system may adjust the imaging device byincreasing the collimation area by actuating collimation blades of theimaging device. Further, in some examples, in response to a dissatisfiedimage quality criteria, wherein the image quality criteria indicates adesired point in time during an inspiration/expiration cycle, or whereinthe image quality criteria indicates motion induced blurring in a lungimage, the imaging system may display via a display device a suggestionfor the imaging subject to hold their breath (either at a state ofinspiration or expiration based on the imaging protocol being employed).In some examples, operation 118 may include the medical imaging systemchanging the mA's of the imaging device to achieve more penetration,higher dose. In some examples, operation 118 may include the medicalimaging system activating AutoGrid (a software processing to removescatter). In some examples, operation 118 may include the medicalimaging system adjusting SID (to avoid anatomical cut off if thecollimator blades are open). In some examples, operation 118 may includethe medical imaging system adjusting collimator blades (shrinking ifthere is a lot of air around the anatomy, or expanding if the region ofinterest is cut off). In some examples, operation 118 may include themedical imaging system turning on grid line reduction (if grid lines aredetected). In some examples, operation 118 may include the medicalimaging system conducting a second image processing called QuickEnhance. In some examples, operation 118 may include the medical imagingsystem moving an OTS tube up/down, left/right if the patient hasangulation or rotation respectively, (moving the system around thepatient, vs asking the patient to reposition). Following operation 118,method 100 may end.

In this way, method 100 enables automatic and rapid determination of oneor more image quality metrics following acquisition of a medical image.In this way, a technician may receive quantitative information regardingone or more geometrical aspects of a medical image with little to nolatency, facilitating the technician's evaluation of, and choice toaccept or reject, the medical image. Further, by storing the imagequality metric as metadata of the medical image, each stakeholder orevaluator may receive a consistent, quantitative, and precise measure ofthe image quality metrics pertaining to a particular image acquired viaa particular imaging protocol.

Referring to FIG. 2 , an imaging system 200 is shown, in accordance withan exemplary embodiment. In some embodiments, at least a portion ofimaging system 200 is disposed at a remote device (e.g., edge device,server, etc.) communicably coupled to imaging system 200 via wiredand/or wireless connections. In some embodiments, at least a portion ofimaging system 200 is disposed at a separate device (e.g., aworkstation) which can receive images from the imaging system 200 orfrom a storage device which stores the images generated by one or moreadditional imaging systems. Imaging system 200 comprises imageprocessing device 202, display device 230, user input device 240, andimaging device 250.

Image processing device 202 includes a processor 204 configured toexecute machine readable instructions stored in non-transitory memory206. Processor 204 may be single core or multi-core, and the programsexecuted thereon may be configured for parallel or distributedprocessing. In some embodiments, the processor 204 may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. In some embodiments, one or more aspects of theprocessor 204 may be virtualized and executed by remotely-accessiblenetworked computing devices configured in a cloud computingconfiguration.

Non-transitory memory 206 may store deep neural network module 208,image quality metric module 210, image quality criteria module 212, andimaging protocol module 214. Deep neural network module 208 may includeone or more deep neural networks, comprising a plurality of weights andbiases, activation functions, and instructions for implementing the oneor more deep neural networks to receive medical images and map themedical images to one or more positional attributes of one or moreanatomical features in the medical images. For example, deep neuralnetwork module 208 may store instructions for implementing a neuralnetwork, such as the exemplary deep neural networks shown in FIGS. 9 and10 . Deep neural network module 208 may include trained and/or untrainedneural networks and may further include various metadata for the one ormore trained or untrained deep neural networks stored therein. In someembodiments, deep neural networks may be stored in locations ofnon-transitory memory indexed according to a value or key, wherein thevalue or key indicates one or more imaging protocols in which the deepneural networks may be employed.

Non-transitory memory 206 may further include image quality metricmodule 210, which comprises instructions for determining one or moreimage quality metrics based on at least a first positional attribute ofan anatomical feature. Image quality metric module 210 may includeinstructions that, when executed by processor 204, cause imageprocessing device 202 to conduct one or more of the steps of methods300, 400, and/or 500, discussed in more detail below with reference toFIGS. 3, 4, and 5 , respectively. In one example, image quality metricmodule 210 includes instructions . . . . In some embodiments, the imagequality metric module 210 is not disposed at the imaging device 200, butis located remotely and communicatively coupled to imaging system 200.

Non-transitory memory 206 may further store image quality criteriamodule 212, wherein a plurality of image quality criteria, associatedwith one or more imaging protocols may be stored.

In some embodiments, the non-transitory memory 206 may includecomponents disposed at two or more devices, which may be remotelylocated and/or configured for coordinated processing. In someembodiments, one or more aspects of the non-transitory memory 206 mayinclude remotely-accessible networked storage devices configured in acloud computing configuration.

Imaging system 200 may further include user input device 240. User inputdevice 240 may comprise one or more of a touchscreen, a keyboard, amouse, a trackpad, a motion sensing camera, or other device configuredto enable a user to interact with and manipulate data within imageprocessing device 202. In one example, user input device 240 may enablea user to annotate an object class of interest in a 3D medical image.

Display device 230 may include one or more display devices utilizingvirtually any type of technology. Display device 230 may be combinedwith processor 204, non-transitory memory 206, and/or user input device240 in a shared enclosure, or may be peripheral display device and maycomprise a monitor, touchscreen, projector, or other display deviceknown in the art, which may enable a user to view 2D medical images, 3Dmedical images, pseudo-3D medical images, and thickness heat-maps,and/or interact with various data stored in non-transitory memory 206.

Imaging system 200 further includes imaging device 250. Imaging device250 may comprise a 2D or 3D medical imaging device, including but notlimited to an x-ray imaging device, a CT imaging device, an MRI system,an ultrasound, and a PET imaging device. Images acquired by imagingdevice 250 may be stored at image data 212 in non-transitory memory 206,or may be stored remotely at an external storage device communicativelycoupled to imaging system 200.

It should be understood that image processing system 200 shown in FIG. 2is for illustration, not for limitation. Another appropriate imageprocessing system may include more, fewer, or different components.

Turning to FIG. 3 , an embodiment of a method 300 for automaticallydetermining a first class of image quality metrics, which may be used toassess an image quality criterion, is shown. Method 300 may be executedby a medical imaging system, such as medical imaging system 200, byexecuting machine readable instructions stored in non-transitory memory.In some embodiments, instructions for executing one or more operationsof method 300 may be stored in an image quality metric module, such asimage quality metric module 212 shown in FIG. 2 . One or more of theoperations of method 300 may be executed by a medical imaging system aspart of another method. In some embodiments, one or more of theoperations of method 300 may be executed as part of method 100,discussed above.

At operation 302, the medical imaging system determines a first locationof a first anatomical feature in a medical image of an imaging subject.The first location is a positional attribute of the first anatomicalfeature, which may be determined by a trained deep neural network. Insome embodiments, the medical imaging system may determine the firstlocation of the first anatomical feature by feeding the medical imageinto a trained deep neural network, wherein the deep neural network mapsthe input medical image to coordinates of the first location using aplurality of learned parameters. In some embodiments, the medicalimaging system feeds the medical image to a trained deep neural networkconfigured to map the medical image to a segmentation mask of the firstanatomical feature, and the first location of the first anatomicalfeature is determined based on the segmentation mask. In someembodiments, the first location may be determined from a segmentationmask by calculating a center-point of the segmentation mask. In someembodiments, the first location of the first anatomical feature may bedetermined via an additional trained deep neural network, configured todetermine for a particular anatomical feature's segmentation mask, apre-determined position or anatomical landmark of said anatomicalfeature. As an example, a deep neural network may be configured todetermine a location of a pupil from a segmentation mask of eyes,wherein eyes are the first anatomical feature in the present example,and wherein the center of the pupil is the first location. In someembodiments, the first location may be determined from a segmentationmask of the first anatomical feature by fitting a bounding box (or othershape), to the point cloud of the segmentation mask, wherein the firstlocation is given at a fixed position relative to the bounding box(e.g., the first position is a function of the bounding box's position),thereby fitting of the bounding box to the segmentation mask implicitlyidentifies the first location. It will be appreciated that operation302, as well as the other operations of method 300, may be applied to 2Dor 3D images.

At operation 304, the medical imaging system determines a secondlocation of a second anatomical feature in the medical image. Themedical imaging system may determine the second location of the secondanatomical feature in a manner analogous to that described above, withreference to operation 302. In some embodiments, a same trained deepneural network may be used to perform the determination of the firstlocation and the second location. In some embodiments, a multi-taskingdeep neural network, such as multi-tasking deep neural network 1000, maybe employed at operations 302 and 304 to determine the first locationand the second location, wherein the first location may be determinedvia a first branch of the multi-tasking deep neural network, and whereinthe second location may be determined via a second branch of themulti-tasking deep neural network. Particularly, if the first anatomicalfeature and the second anatomical feature belong to a same anatomicalclass (e.g., teeth, phalanges, etc.), or represent anatomical featuresrelated by symmetry (e.g., a left hand and a right hand). In someembodiments, both the first location and the second location may bedetermined from a single segmentation mask, by natural extension of theapproaches discussed above at operation 302 with regards to segmentationmasks and determination of a single location, to the case of twolocations.

At operation 306, the medical imaging system determines a third locationof a third anatomical feature in the medical image. The medical imagingsystem may determine the third location of the third anatomical featurein a manner analogous to that described above, with reference tooperations 302, and 304. In some embodiments, distinct trained deepneural networks may be employed to determine the first location, thesecond location and the third location. In some embodiments, a singletrained deep neural network may be used to determine two or more of thefirst location, the second location, and the third location. In someembodiments, two or more of the first location, the second location, andthe third location, may be determined from a single segmentation maskproduced by a single trained deep neural network.

At operation 308, the medical imaging system determines a first distancebetween the first location and the second location. In some embodiments,the first location and second location comprise points in 2D or 3Dspace, uniquely identified by two coordinates, or three coordinates (inthe case of 2D and 3D respectively), and the first distance may bedetermined by calculating the Euclidean distance between coordinates ofthe first location and the second location. In some embodiments,operation 308 may include determination of a vector connecting the firstlocation and second location, thereby providing a relative displacementof the first location with respect to the second location. In someembodiments, the distance may be measured in pixels and/or voxels. Insome embodiments, a distance may be determined in physical units, suchas feet or meters.

At operation 310, the medical imaging system determines a seconddistance between the second location and the third location. The medicalimaging system may determine the second location in a manner analogousto that described above, with respect to the first distance determinedat operation 308.

At operation 312, the medical imaging system determines a ratio betweenthe first distance and the second distance.

At operation 314, the medical imaging system determines an angle ofrotation of the imaging subject with respect to the plane of projectionof the medical image. In some embodiments, a difference between thefirst distance and the second distance may be proportional to an angleof rotation of the imaging subject with respect to the projection planof the medical image, such as when the first anatomical feature and thethird anatomical feature are symmetrical anatomical features (e.g., leftand right hands, left and right clavicles, etc.). In some embodiments,one or more trigonometric relationships between the first anatomicalfeature, the second anatomical feature, and the third anatomicalfeature, may be used in conjunction with the first distance and thesecond distance to determine the angle of rotation. In some embodiments,a computational complexity of method 300 may be reduced by expressing animage quality criterion for an angle of rotation in terms of a range ofrelative first distances and second distances, thus enabling directcomparison between a ratio or difference of the currently measured firstdistance and second distance, against a threshold range of distanceratios (or differences) corresponding to the desire angular range. Thismay reduce the need for additional calculations converting the first andsecond distance into an angle of rotation at the time of imageacquisition.

At operation 316, the medical imaging system compares the angle ofrotation against a pre-determined rotation range. The medical imagingsystem may access a value, or range of values, corresponding to thedesired angular range, and may determine if the angle (or values)determined at operation 314 fall with, or satisfy, the value or values.In some embodiments, an image quality criterion may comprise a thresholdrange of rotation of an imaging subject with respect to a projectionplane of a medical image, wherein the image quality metric may comprisea currently determined angle, and the image quality criterion maycomprise a pre-determined threshold degree of rotation, or an upper andlower angular threshold, wherein if the currently determined angle ofrotation exceeds the upper angular threshold, or is less than the lowerangular threshold, the medical imaging system responds by setting astatus of the image quality criteria to a value indicating the imagequality metric of the medical image does not satisfy the image qualitycriterion. Conversely, if the image quality metric determined atoperation 314 satisfies the pre-determined rotation range, the medicalimaging system responds by setting a status of the rotation imagequality criterion to a value indicating the image quality metricsatisfies the image quality criterion.

At operation 318, the medical imaging system displays the medical imageand at least one of the first distance, the second distance, the angleof rotation, and the pre-determined rotation range. By displaying theimage quality metric (the angle of rotation) along with thepre-determined rotation range, a technician or radiologist may quicklyevaluate a quantitative comparison between the actual rotation angle ofthe medical image and a standard or desired range of rotation angles.Following operation 318, method 300 may end.

Turning briefly to FIG. 6 , an exemplary embodiment of a GUI 600, whichmay be displayed as part of an automated image quality assessment, suchas at operation 318 of method 300, is shown. GUI 600 includes a medicalimage 650, comprising a chest x-ray of an imaging subject. Superimposedon medical image 650 is a first distance 608, between a first location602, located on a first anatomical feature 612 (a right clavicle) and asecond location 606, of a second anatomical feature 614 (a left clavicleof the imaging subject). Further GUI 600 shows a second distance 610,between the second location 606 and the third location 604, located on athird anatomical feature 616 (the spinous process). Numerical values forboth the first distance 608, and the second distance 610, are shownsuperimposed on medical image 650, proximal to the first distance 608and second distance 610. Units of the first distance 608 and the seconddistance 610 are in pixels. Further, the first anatomical feature 612,the second anatomical feature 614, and the third anatomical feature 616,are shown in highlight, emphasizing the accuracy of the segmentationmasks of the first, second, and third anatomical features, respectively.Thus, GUI 600 provides quantitative image quality metrics at a glance,as well as indications of the positional attributes of the anatomicalfeatures upon from which the image quality metric was determined. Byproviding this quantitative information for an image quality metric, GUI600 enables a technician or radiologist to more quickly, accurately, andconsistently, evaluate geometrical aspects of a medical image, which mayotherwise be difficult, slow, or impossible, to ascertain by eye.

Turning to FIG. 4 , an embodiment of a method 400 for automaticallydetermining a second class of image quality metric, which may be used toassess an image quality criterion, is shown. Method 400 may be executedby a medical imaging system, such as medical imaging system 200, byexecuting machine readable instructions stored in non-transitory memory.In some embodiments, instructions for executing one or more operationsof method 400 may be stored in an image quality metric module, such asimage quality metric module 212 shown in FIG. 2 . One or more of theoperations of method 400 may be executed by a medical imaging system aspart of another method. In some embodiments, one or more of theoperations of method 400 may be executed as part of method 100,discussed above.

At operation 402, the medical imaging system maps a medical image of animaging subject to a first segmentation mask of a first anatomicalfeature. In some embodiments, the segmentation mask may comprise a 2Dsegmentation mask, or 3D segmentation mask. The medical imaging systemmay access a pre-loaded deep neural network, wherein said deep neuralnetwork is trained to predict segmentation masks for the firstanatomical feature. An exemplary approach for mapping a medical image toa segmentation mask is given by deep neural network architecture 900,discussed below in the description of FIG. 9 . Briefly, segmentationmasks may comprise an array or matrix of values, corresponding to thearray or matrix of pixel/voxel intensity values of an input image. Eachvalue may be associated with a region of the medical image correspondingto a pixel or voxel. Values may be of one or more discrete andpre-determined numbers or IDs, uniquely matching to a set of anatomicalclass labels, indicating if the corresponding pixel or voxel has beenclassified as a member of said one or more classes.

At operation 404, the medical imaging system maps the medical image to asecond segmentation mask of a second anatomical feature. The medicalimaging system may map the medical image to the second segmentation maskin a manner substantially analogous to that described above, withrespect to operation 402. In some embodiments, a same trained deepneural network may be used to perform the determination of the firstsegmentation mask and the second segmentation mask. In some embodiments,a multi-tasking deep neural network, such as multi-tasking deep neuralnetwork 1000, may be employed at operations 402 and 404 to determine thefirst segmentation mask and the second segmentation mask, wherein thefirst segmentation mask may be determined via a first branch of themulti-tasking deep neural network, and wherein the second segmentationmask may be determined via a second branch of the multi-tasking deepneural network. Particularly, if the first anatomical feature and thesecond anatomical feature belong to a same anatomical class (e.g.,teeth, phalanges, etc.), or represent anatomical features related bysymmetry (e.g., a left hand and a right hand), a multi-tasking deepneural network may be particularly advantageous, as a shared encodedfeature map may be used in determination of both the first segmentationmask and the second segmentation mask. In some embodiments, both thefirst segmentation mask and the second segmentation mask may bedetermined from a single deep neural network, wherein the deep neuralnetwork is not a multi-tasking deep neural network. In some embodiments,the first segmentation mask may be produced by a first deep neuralnetwork, and the second segmentation mask may be produced by a seconddeep neural network, wherein the first deep neural network is distinctand shares no layers or parameters with the first deep neural network.

At operation 406, the medical imaging system determines an extent ofoverlap between the first anatomical feature and the second anatomicalfeature based on the extent of intersection between the firstsegmentation mask and the second segmentation mask. In some embodiments,both the first and second segmentation masks occur in a same coordinatesystem, wherein a first point or pixel from a medical image maycorrespond to a second point in the first segmentation mask, and a thirdpoint in the second segmentation mask, wherein the second point and thethird point occur at a same coordinate address in their respectivesegmentation masks. In other words, a point at (1,1) in a firstsegmentation mask corresponds to a point at (1,1) in the secondsegmentation mask. Thus, the medical imaging system may determine theintersection/overlap of the first feature with the second feature, basedon the number of points in the first segmentation mask classified asbelonging to the first anatomical feature, which have matching points(that is, points occurring at the same coordinate address) in the secondsegmentation mask classified as belonging to the second anatomicalfeature. Thus, the area (or volume) of intersection between the firstanatomical feature and the second anatomical feature may be determinedby multiplying said number of matching points between the firstsegmentation mask and the second segmentation mask by a proportionalityconstant, giving the spatial area (or volume) occupied by each pixel orvoxel in the medical image.

At operation 408, the medical imaging system compares the extent ofoverlap between the first anatomical feature and the second anatomicalfeature with a threshold extent of overlap. The medical imaging systemmay access a value, or range of values, indicating a desired extent ofoverlap, wherein said value or values may be indexed according toimaging protocol, thereby enabling rapid access to the value or valuesin response to receiving an imaging protocol selection. In someembodiments, the value or values may be stored in units of overlappingsegmentation mask points, as opposed to spatial extents of overlap, thusreducing the computational expense of converting intersecting points toa spatial area or volume. In some embodiments, the threshold extent ofoverlap comprises a value, indicating an upper, or a lower, limit ofdesired overlap. As an example, an upper threshold extent of overlap maybe set to 400 cm², and medical images including an extent of overlap ofgreater than 400 cm², are considered as not satisfying or meeting theimage quality criteria of 400 cm², whereas medical images having anextent of overlap less than 400 cm² are considered as satisfying theimage quality criteria.

At operation 410, the medical imaging system displays the medical imageand at least one of the first segmentation mask, the second segmentationmask, an area of intersection between the first segmentation mask andthe second segmentation mask, and a status indicating if the extent ofoverlap satisfies the threshold extent of overlap. Following operation410, method 400 may end. In this way, method 400 enables a technician orradiologist to quickly determine if a first anatomical feature isoccluding a second anatomical feature, in a medical image. This mayenable a technician to quickly determine that greater than a desiredextent of overlap or occlusion is present in a medical image, in caseswhere such occlusion is undesired. Conversely, in some imagingprotocols, alignment between a first anatomical feature and a secondanatomical feature may be desired. In such cases, method 400 enables atechnician or radiologist to quickly determine if a desired extent ofoverlap is achieved.

Turning to FIG. 7 , a graphical user interface 700, which may bedisplayed as part of an automated image quality assessment, such as atoperation 410 of method 400, is shown. User interface 700 showssegmentation map 702, indicating the automatically inferred regionoccupied by the C1 vertebrate of the imaging subject. As can be theseen, the outline of the C1 vertebrae as determined by the segmentationmap 702 is fully visible, that is, does not overlap with the teeth ofthe imaging subject. By showing a segmentation map, or heat map, of theposition of an anatomical feature of interest, a radiologist or imagingtechnician may be enabled to quickly assess if the anatomy of interestis clipped or blocked. In some embodiments, a quantitative image qualitymetric, such as a visibility score indicating an automatically inferredextent of clipping of an anatomical region of interest may be displayedin GUI 700, wherein an anatomical feature with no clipping or blockagemay be given a high visibility score, and an anatomical region ofinterest with substantial blocking or clipping may be given a lowvisibility score. The visibility score may be superimposed on themedical image proximal to the segmentation map 702.

Turning to FIG. 5 , an embodiment of a method 500 for automaticallydetermining a second image quality metric, which may be used to assessan image quality criterion, is shown. Method 500 may be executed by amedical imaging system, such as medical imaging system 200, by executingmachine readable instructions stored in non-transitory memory. In someembodiments, instructions for executing one or more operations of method500 may be stored in an image quality metric module, such as imagequality metric module 212 shown in FIG. 2 . One or more of theoperations of method 500 may be executed by a medical imaging system aspart of another method. In some embodiments, one or more of theoperations of method 500 may be executed as part of method 100,discussed above.

At operation 502, the medical imaging system maps a medical image of animaging subject to a classification score for an anatomical feature. Insome embodiments, the classification score may comprise a probability orconfidence of the anatomical feature being present and observable withina field-of-view of the medical image. In some embodiments, theclassification score may comprise a binary label, indicating either apresence or absence of the anatomical feature from the field-of-view ofthe medical image. Method 500 is described with reference to a singleanatomical feature, however it will be appreciated that method 500 maybe extended to a plurality of anatomical features, wherein an anatomicalscore for each of the plurality of anatomical features may bedetermined.

At operation 504, the medical imaging system compare the classificationscore against a classification score threshold. The classification scorethreshold may comprise a single value (in the case of a binary,true/false, classification score), or a range of values, in the case ofa real valued classification score. In some embodiments, classificationscore threshold may be stored in locations of non-transitory memoryassociated with one or more imaging protocols, wherein, in response toreceiving an imaging protocol selection, the medical imaging system mayaccess the image quality criteria indicated by the imaging protocolselection.

At operation 506, the medical imaging system display the medical image,and at least one of the classification score, and the classificationscore threshold via a display device. Following operation 506, method500 may end. Two exemplary embodiments of GUIs which may be displayed atoperation 506 are shown in FIGS. 8A an 8B.

Turning to FIGS. 8A and 8B, GUI 800A and GUI 800B, which may bedisplayed as part of an automated image quality assessment, such as atoperation 506 of method 500, are shown. GUI 800A includes a firstmedical image 804A, showing a view of the cervical vertebra of animaging subject. GUI 800A further includes a table 802A, showing aplurality of classification scores for a plurality of anatomicalfeatures. In particular, table 802A includes a first row, indicatingeach of a plurality of anatomical features, from the C1 vertebra to theT1 vertebra. Below each anatomical feature label in table 802A, is astatus of an image quality criteria assessment, wherein anatomicalfeatures with classification scores above a corresponding classificationscore threshold, are indicated via a check mark, whereas anatomicalfeatures with classification scores below a corresponding classificationscore threshold are indicated with an X. As can be seen in FIG. 8A, thefirst medical image 804A has satisfied each of the classification scoreimage quality criteria, as each of C1-T1 are visible in thefield-of-view of first medical image 804A. Conversely, in FIG. 8B, table802B indicates the classification thresholds are satisfied foranatomical features C1-C6, but are not satisfied for anatomical featuresC7 and T1. In other words, second medical image 804B has a lowprobability of the C7-T1 vertebra being visible in the field-of-view.

Turning to FIG. 9 , a deep neural network architecture 900 for mappingmedical images to positional attributes of an anatomical feature ofinterest is shown, in accordance with an exemplary embodiment. Inparticular, deep neural network architecture 900 is a convolutionalneural network (CNN) architecture, configured to map medical images tosegmentation masks for one or more anatomical features of interest. Deepneural network architecture 900 is configured to receive an image tile,comprising pixel/voxel intensity values in one or more color channels,extract and embed features from the input image tile, and map saidencoded features to a segmentation mask. From the segmentation mask,information regarding a size, location, orientation, or other positionalattribute of the anatomical features may be determined. Deep neuralnetwork architecture 900 includes a series of mappings, from an inputimage tile 902 which may be received by an input layer, through aplurality of feature maps, and finally to an output segmentation mask956 which may be produced by an output layer. Although input into deepneural network architecture 900 is referred to herein as comprisingpixel/voxel intensity tiles, it will be appreciated that additional ordifferent features may be fed into deep neural networks disclosedherein. In one embodiment, additional features, such as patient/imagingsubject specific information, may be concatenated with an input imagetile and fed into deep neural network architecture 900.

The layers and operations/transformations comprising deep neural networkarchitecture 900 are labeled in legend 958. As indicated by legend 958,deep neural network architecture 900 includes a plurality of featuremaps, wherein each feature map may be produced by applying atransformation or mapping to one or more previous feature maps (or inputdata in the case of the input image tile 902). Each feature map maycomprise a multi-dimensional matrix, or multi-dimensional array, offeature values, wherein each feature value may be uniquely identified bya set of N_(i) indices, wherein N_(i) is the number of dimensions in thei^(th) feature map. The size of a feature map may be described usingspatial dimensions. As an example, length, width, and depth, may be usedto refer to the number of rows, columns, and channels, in athree-dimensional feature map. For feature maps of N_(i) greater thanthree, terms such as hyper-width, hyper-depth, and hyper-length, may beused.

The transformations/mappings performed on each feature map are indicatedby arrows, wherein each type of arrow corresponds to a uniquetransformation, as indicated by legend 958. Rightward pointing solidblack arrows indicate 3×3 convolutions and activations, wherein afeature value for an i^(th) feature map is determined by calculating adot product between a 3×3×j_(i-1) filter and a 3×3×j_(i-1) group offeature values from the i-1^(th) feature map, wherein j_(i-1) is thenumber of feature channels of the i-1^(th) feature map. The dot productis passed through a pre-determined activation function to determine thefeature value for the i^(th) feature map.

Downward pointing arrows indicate 2×2 max pooling, wherein the max valuefrom a 2×2×1 group of feature values from an i-1^(th) feature map ispropagated to an i^(th) feature map, thereby resulting in a 4-foldreduction in spatial resolution of the i^(th) feature map compared tothe i-1^(th) feature map. In some embodiments, each feature channel ispooled separately, thus conserving the number of feature channelsbetween the i and i-1^(th) feature maps.

Upward pointing arrows indicate 2×2 up convolutions, wherein output froma single feature channel of an feature map is mapped to a 2×2 grid offeature values in an i^(th) feature map, thereby resulting in a 4-foldincrease in spatial resolution of the i^(th) feature map compared to thei-1^(th) feature map.

Rightward pointing dash-tailed arrows indicate copying and cropping ani-m^(th) feature map and concatenating the copied feature map to ani^(th) feature map, wherein m may be a function of i. Cropping enablesthe dimensions of the i-m^(th) feature map (excluding the channel depth)to match the dimensions of the i^(th) feature map. Cropping andconcatenating increases the feature channel depth of the i^(th) featuremap.

Rightward pointing arrows with hollow heads indicate a 1×1×j_(i-1)convolution and activation, wherein a dot product is determined betweena 1×1×j_(i-1) group of feature values of the i-1^(th) feature map, and a1×1×j_(i-1) filter, wherein j_(i-1) is the number of feature channels ofthe i-1^(th) feature map. The dot product may be passed through anactivation function to produce a feature value for the i^(th) featuremap. The 1×1 convolution and activation does not change the spatialresolution of the input feature map, as there is a 1-to-1 mappingbetween each spatially distinct feature in the input feature map andeach spatially distinct feature in the output feature map.

In addition to the operations indicated by the arrows within legend 958,Deep neural network architecture 900 includes solid filled rectanglescorresponding to feature maps, wherein feature maps comprise a height(top to bottom length as shown in FIG. 9 , corresponds to a y spatialdimension in an x-y plane), width (not shown in FIG. 9 , assumed equalin magnitude to height, corresponds to an x spatial dimension in an x-yplane), and depth (a left-right length as shown in FIG. 9 , correspondsto the number of feature channels within each feature map). Likewise,Deep neural network architecture 900 includes hollow (unfilled)rectangles, corresponding to copied and cropped feature maps, whereincopied feature maps comprise height (top to bottom length as shown inFIG. 9 , corresponds to a y spatial dimension in an x-y plane), width(not shown in FIG. 9 , assumed equal in magnitude to height, correspondsto an x spatial dimension in an x-y plane), and depth (a length from aleft side to a right side as shown in FIG. 9 , corresponds to the numberof feature channels within each feature map).

Starting at input image tile 902 (herein also referred to as an inputlayer), data corresponding to a medical image may be input and mapped toa first feature map. In some embodiments, the input data correspondsgrayscale pixel/voxel intensity values. In some embodiments, the inputdata corresponds to pixel/voxel intensity values in a plurality of colorchannels. The input data may correspond to two-dimensional (2D) orthree-dimensional (3D) medical images. In some embodiments, the inputdata is pre-processed (e.g., normalized) before being processed by deepneural network architecture 900.

Take a specific configuration as an example for the purpose ofillustration. Input image tile 902 includes a feature map comprising572×572×K feature values, corresponding to pixel intensity values of a572×572 2D medical image having K color channels. In some embodiments, Kmay be greater than one, wherein the input image tile comprises aseparate feature channel for each of the K color channels. For example,in an RGB pixel color model, K may be three, and the input image tile902 may comprise 572×572 intensity values per each of the three colorchannels, for a total of 572×572×3 input values/features. In someembodiments, K may be one, such as in a a grayscale/black-and-whitecolor scheme.

As indicated by the solid black rightward pointing arrow immediately tothe right of input image tile 902, a 3×3×K convolution of the inputimage tile 902 is performed to produce feature map 904. As discussedabove, a 3×3 convolution includes mapping a 3×3×j_(i-1) group of featurevalues from an i-1^(th) feature map to a single feature value of ani^(th) feature map using a 3×3×j_(i-1) convolutional filter. For eachdistinct convolutional filter applied to the i-1^(th) feature map, afeature channel is added to the i^(th) feature map, thus the number ofdistinct filters applied to an input feature map corresponds to thenumber of feature channels in the output feature map. In deep neuralnetwork architecture 900, 64 distinct filters are applied to the inputimage tile 902, thereby generating feature map 904, comprising 64feature channels. Each of the 64 distinct filters comprise a distinctgroup of learned weights, with a fixed positional relationship withrespect to each other filter weight in the group. The increase infeature channels between input image tile 902 and feature map 904 isindicated by an increase in the left-right width of feature map 904compared to input image tile 902. The 3×3 convolutions of deep neuralnetwork architecture 900 comprise step sizes of 1, and therefore resultin a loss of a 1 pixel border from the input image for each 3×3convolution applied. Therefore, feature map 904 includes a spatialresolution of 570×570 (that is, two feature channels are lost in the xdimension and two feature channels are lost in the y dimension).

Feature map 904 includes 570×570×64 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 904, a 3×3 convolution is performed on feature map 904 to producefeature map 906.

Feature map 906 includes 568×568×64 feature values. As indicated by thedownward pointing arrow beneath feature map 906, a 2×2 max poolingoperation is performed on feature map 906 to produce feature map 908.Briefly, a 2×2 max pooling operation includes determining a max featurevalue from a 2×2 grid of feature values from a single feature channel animmediately preceding feature map, and setting a single feature value,in a single feature channel, of a current feature map, to the max valueso determined. The 2×2 max pooling employed herein includes a step sizeof two. 2×2 max pooling thereby combines output from 4 feature values (2in the x dimension and 2 in they dimension) to produce a reduced spatialresolution feature map (the output feature map will comprise half thenumber of feature values in the x direction and half the number offeature values in the y direction. Or said another way, the outputfeature map will comprise one fourth the number of feature values (perfeature channel) compared to the input feature map). 2×2 max poolingdoes not alter the number of feature channels, as pooling is appliedseparately to each distinct feature channel of the input feature map,e.g., features between multiple feature channels are not combined.Additionally, a copy of feature map 906 is cropped and concatenated withoutput from feature map 948 to produce feature map 950, as indicated bythe dash-tailed rightward pointing arrow immediately to the right offeature map 906.

Feature map 908 includes 284×284×64 feature values (a fourth the spatialresolution of feature map 906, due to the 2×2 max pooling) with 64features per channel. As indicated by the solid black rightward pointingarrow immediately to the right of feature map 908, a 3×3 convolution isperformed on feature map 908 to produce feature map 910.

Feature map 910 includes 282×282×128 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 910, a 3×3 convolution is performed on feature map 910 to producefeature map 912.

Feature map 912 includes 280×280×128 feature values. As indicated by thedownward pointing arrow beneath feature map 912, a 2×2 max poolingoperation is performed on feature map 912 to produce feature map 914,wherein feature map 914 is of one fourth the spatial resolution offeature map 912. Additionally, feature map 912 is cropped, copied, andconcatenated with output from feature map 942 to produce feature map944, as indicated by the dash-tailed rightward pointing arrowimmediately to the right of feature map 912.

Feature map 914 includes 140×140×128 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 914, a 3×3 convolution is performed on feature map 914 to producefeature map 916.

Feature map 916 includes 198×198×256 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 916, a 3×3 convolution is performed on feature map 916 to producefeature map 918.

Feature map 918 includes 196×196×256 feature values. As indicated by thedownward pointing arrow beneath feature map 918, a 2×2 max poolingoperation is performed on feature map 918 to produce feature map 920,wherein feature map 920 is of one fourth the spatial resolution offeature map 918. Additionally, feature map 918 is cropped, copied, andconcatenated with output from feature map 936 to produce feature map938, as indicated by the dash-tailed rightward pointing arrowimmediately to the right of feature map 918.

Feature map 920 includes 68×68×256 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 920, a 3×3 convolution is performed on feature map 920 to producefeature map 922.

Feature map 922 includes 66×66×512 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 922, a 3×3 convolution is performed on feature map 922 to producefeature map 924.

Feature map 924 includes 64×64×512 feature values. As indicated by thedownward pointing arrow beneath feature map 924, a 2×2 max poolingoperation is performed on feature map 924 to produce feature map 926,wherein feature map 926 is of one fourth the spatial resolution offeature map 924. Additionally, feature map 924 is cropped, copied, andconcatenated with output from feature map 930 to produce feature map932, as indicated by the dash-tailed rightward pointing arrowimmediately to the right of feature map 924.

Feature map 926 includes 925×925×512 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 926, a 3×3 convolution is performed on feature map 926 to producefeature map 928.

Feature map 928 includes 90×90×1024 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 928, a 3×3 convolution is performed on feature map 928 to producefeature map 930.

Feature map 930 includes 28×28×1024 feature values. As indicated by theupward pointing arrow immediately above feature map 930, a 2×2up-convolution is performed on feature map 930 to produce a firstportion of feature map 932, while copied and cropped feature values fromfeature map 924 are used to produce a second portion of feature map 932.Briefly, a 2×2 up-convolution includes mapping a 1×1×j_(i-1) group offeature values in an input feature map to a 2×2×1 group of featurevalues in a current feature map using a 2×2×j_(i-1) filter (that is,features corresponding to each feature channel at a single spatialposition of an input feature map are mapped to four spatial positions ofa single feature channel of the output feature map). For each distinctupconvolutional filter applied to an input feature map, a single featurechannel is produced in an output feature map. In the upconvolution offeature map 930, 512 distinct upconvolutional filter are applied, and anadditional 512 feature channels are added from the copied and croppedfeature map 924, to produce feature map 932 with 1024 feature channels.

Feature map 932 includes 56×56×1024 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 932, a 3×3 convolution is performed using 512 distinct convolutionalfilters on feature map 932 to produce feature map 934.

Feature map 934 includes 54×54×512 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 934, a 3×3 convolution is performed on feature map 934 using 512distinct convolutional filters to produce feature map 936.

Feature map 936 includes 52×52×512 feature values. As indicated by theupward pointing arrow immediately above feature map 936, a 2×2upconvolution is performed using 256 distinct up convolutional filterson feature map 936 to produce a first portion of feature map 938, whilecopied and cropped features from feature map 918 produce a secondportion of feature map 938.

Feature map 938 includes 104×104×512 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 938, a 3×3 convolution is performed on feature map 938 using 256distinct convolutional filters to produce feature map 940.

Feature map 940 includes 102×102×256 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 940, a 3×3 convolution is performed on feature map 940 using 256distinct convolutional filters to produce feature map 942.

Feature map 942 includes 100×100×256 feature values. As indicated by theupward pointing arrow immediately above feature map 942, a 2×2upconvolution is performed on feature map 942 using 128 distinctupconvolutional filters to produce a first portion of feature map 944,while copied and cropped features from feature map 912 are used toproduce the second portion of feature map 944.

Feature map 944 includes 200×200×256 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 944, a 3×3 convolution is performed on feature map 944 using 128distinct convolutional filters to produce feature map 946.

Feature map 946 includes 198×198×128 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 946, a 3×3 convolution is performed on feature map 946 using 128distinct convolutional filters to produce feature map 948.

Feature map 948 includes 196×196×128 feature values. As indicated by theupward pointing arrow immediately above feature map 948, a 2×2upconvolution is performed on feature map 948 using 64 distinctconvolutional filters to produce a first portion of feature map 950,while copied and cropped features from feature map 906 are used toproduce a second portion of feature map 950.

Feature map 950 includes 392×392×128 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 950, a 3×3 convolution is performed on feature map 950 using 64distinct convolutional filters to produce feature map 952.

Feature map 952 includes 390×390×64 feature values. As indicated by thesolid black rightward pointing arrow immediately to the right of featuremap 952, a 3×3 convolution is performed on feature map 952 using 64distinct convolutional filters to produce feature map 954.

Feature map 954 includes 388×388×64 feature values. As indicated by thehollow headed rightward pointing arrow immediately to the right offeature map 954, a 1×1 convolution is performed on feature map 954 usingP distinct convolutional filters, to produce segmentation masks 956,wherein P is the number of distinct segmentation masks to be output bydeep neural network architecture 900. Briefly, a 1×1 convolutioncomprises determining a dot product between a 1×1×j_(i-1) group offeature values from an i-1^(th) feature map and a 1×1×j_(i-1)convolutional filter, wherein j_(i-1) is a number of feature channels inthe i-1^(th) feature map. The dot product may then be passed through anactivation function to produce a feature value for an i^(th) featuremap.

Segmentation masks 956 include 388×388×P feature values, wherein P isthe number of distinct segmentation masks produced. In some embodiments,a distinct segmentation mask may be determined for each of P distinctanatomical features from a single input image tile in a single passthrough deep neural network architecture 900. The series ofconvolutions, pooling, activations, and upconvolutions, therefore resultin a reduction of resolution of the input image tile 902 from 572×572pixels to an output segmentation mask resolution of 388×388 pixels.Segmentation masks 956 may comprise a plurality (P) of matrices, eachmatrix comprising a plurality of values indicating a classification of acorresponding pixel of input image tile 902 for a particular anatomicalfeature. In one example, a first segmentation mask comprising 388×388×1values, may indicate an area of a medical image occupied by a firstanatomical feature, wherein the 388×388×1 values may indicate aprobability of a corresponding pixel belonging to the first anatomicalfeature of interest. In some embodiments, the segmentation mask may bebinary, and pixels identified as belonging to an anatomical feature ofinterest may be set to a first value, while pixels identified as notbelonging to the anatomical feature of interest may be set to a secondvalue.

In this way, deep neural network architecture 900 may enablesegmentation/classification of a plurality of pixels/voxels of a medicalimage.

It should be understood that the architecture and configuration of CNN900 shown in FIG. 9 is for illustration, not for limitation. Anyappropriate neural network can be used herein for segmenting medicalimages and/or determining other positional attributes of input medicalimages, such as ResNet, autoencoder, recurrent neural networks, GeneralRegression Neural Network (GRNN), etc.

Turning to FIG. 10 , an exemplary embodiment of a multi-tasking deepneural network 1000 is shown. Multi-tasking deep neural network 1000 maybe implemented by a medical imaging system to rapidly map a medicalimage to a plurality of distinct positional attributes for an anatomicalfeature, wherein the plurality of positional attributes may be used inaccordance with one or more of the methods disclosed herein to determineimage quality metrics. In some embodiments, multi-tasking deep neuralnetwork 1000 may be stored in memory of an image processing system, suchas in deep neural network module 208 of image processing system 202shown in FIG. 2 . Multi-tasking deep neural network 1000 may be indexedin memory based on imaging protocol, thereby linking a location inmemory where multi-tasking deep neural network 1000 is stored, and oneor more imaging protocols for which multi-tasking deep neural network1000 may be employed to determine one or more image quality metrics. Inthis way, multi-tasking deep neural network 1000 may be rapidly loadedfrom memory, which may reduce image assessment latency during an imagingprocess.

Multi-tasking deep neural network 1000 comprises a feature encodingnetwork 1004, which is configured to receive medical images, such asmedical image 1002, and map said medical images to corresponding encodedfeature maps. Said feature maps may comprise a multi-dimensional matrixor array of feature values, wherein said feature values providespatially coherent encoded information extracted from input medicalimage 1002. In some embodiments, the feature encoding network 1004 maycomprise a convolutional neural network, comprising one or moreconvolutional layers, such as are described in more detail above withreference to FIG. 9 .

The encoded feature map produced by feature encoding network 1004 is fedto each of a plurality of branch networks, including first branchnetwork 1006 and N^(th) branch network 1010, wherein each branch networkreceives a copy of the encoded feature map from a shared featureencoding network 1004. FIG. 10 shows only two distinct branch networks,for simplicity, however it will be appreciated that multi-tasking deepneural network 1000 may comprise any positive integer number of branchnetworks greater than one. By sharing an encoded feature map produced bya single feature encoding network 1004, a speed of inference may beincreased in comparison to a plurality of separate deep neural networkswith no shared layers, as a total number of calculations, as well as atotal number of parameters, may be reduced. Further, by sharing afeature encoding network 1004, the multi-tasking deep neural network1000 occupies less memory than N separate networks (wherein N is thetotal number of branch networks), and may be trained in a shorterduration of time, as well as be less susceptible to overfitting.

Each of the plurality of branch networks, including first branch network1006 to N^(th) branch network 1010, may comprise a distinct number,type, and arrangement, of layers. Further, each of the plurality ofbranch networks may comprise a distinct set of parameters (e.g., weightsand biases). In some embodiments, two or more branch networks may sharea common architecture, that is, may have a same number, type, andarrangement of layers, but may comprise distinct parameter values. Eachof the plurality of branch networks may output a distinct positionalattribute, such as first positional attribute 1008 produced by firstbranch network 1006, and N^(th) positional attribute 1012 produced byN^(th) branch network 1010. In some embodiments, first positionalattribute 1008, output by first branch network 1006, may comprise asegmentation map of an anatomical feature, whereas output from N^(th)branch network 1010 may comprise a classification score indicating aprobability that an input medical image includes the anatomical featurein the field of view. The plurality of branch networks may include bothclassification networks, and regression networks, that is, networkstrained to predict a discrete classification label (e.g., predict alabel from the set of labels A, B, and C) and networks trained topredict a real valued number (e.g., predict an area of coverage of lungsshown in a medical image).

When determining a plurality of positional attributes for a singleanatomical feature, computational efficiency and prediction accuracy maybe improved by extracting an encoded representation of said anatomicalfeature using a shared feature encoding network 1004. By sharing acommon feature encoding network for a plurality of downstream predictiontasks for a same anatomical feature, the feature encoding network 1004learns to map medical images to a balanced encoded representation, asthe encoded representation is general enough to inform prediction of adiverse set of positional attributes. Further, as the plurality ofpositional attributes are determined from a common/shared encodedrepresentation, and pertain to a common/shared anatomical feature,coherence between the plurality of positional attributes may beenhanced. As an example, if a first neural network is tasked withdetermining if a medical image contains, or does not contain ananatomical feature, and a second, separate, neural network is taskedwith determining a size of said anatomical feature, it is possible thatthe first neural network determine the anatomical feature is not presentin the medical image, and the second neural network determine that theanatomical feature has a size of 48 pixels. This is incoherence, and mayarise when deep neural networks separately learn to extract an encodedrepresentation of a medical image. The inventors herein have realizedthat by sharing a feature encoding network when determining a pluralityof positional attributes of a single anatomical feature, coherence ofthe positional attributes is improved, which is particularlyadvantageous in determining quantitative image quality metrics.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralof said elements or steps, unless such exclusion is explicitly stated.Furthermore, references to “one embodiment” of the present disclosureare not intended to be interpreted as excluding the existence ofadditional embodiments that also incorporate the recited features.Moreover, unless explicitly stated to the contrary, embodiments“comprising,” “including,” or “having” an element or a plurality ofelements having a particular property may include additional suchelements not having that property. The terms “including” and “in which”are used as the plain-language equivalents of the respective terms“comprising” and “wherein.” Moreover, the terms “first,” “second,” and“third,” etc. are used merely as labels, and are not intended to imposenumerical requirements or a particular positional order on theirobjects.

This written description uses examples to disclose the invention,including the best mode, and also to enable a person of ordinary skillin the relevant art to practice the invention, including making andusing any devices or systems and performing any incorporated methods.The patentable scope of the invention is defined by the claims, and mayinclude other examples that occur to those of ordinary skill in the art.Such other examples are intended to be within the scope of the claims ifthey have structural elements that do not differ from the literallanguage of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

1. A method comprising: acquiring a medical image of an anatomicalregion; mapping the medical image to a positional attribute of ananatomical feature using a trained deep neural network; determining animage quality metric based on the positional attribute of the anatomicalfeature; determining if the image quality metric satisfies an imagequality criterion; and responding to the image quality metric notsatisfying the image quality criterion by: displaying the medical image,the image quality metric, and an indication of the unsatisfied imagequality criterion, via a display device.
 2. The method of claim 1, themethod further comprising: responding to the image quality metricsatisfying the image quality criterion by: displaying the medical image,the image quality metric, and an indication of the satisfied imagequality criterion, via the display device.
 3. The method of claim 1,wherein the positional attribute is one or more of a location of theanatomical feature with respect to the medical image, an orientation ofthe anatomical feature with respect to the medical image, an area of theanatomical feature with respect to the medical image, and aclassification score indicating a probability of inclusion of theanatomical feature within a field of view of the medical image.
 4. Themethod of claim 1, wherein the image quality metric includes one or moreof a costophrenic angle, a cardiothoracic ratio, a mediastinal width, anangle of rotation of an imaging subject with respect to a projectionplane of the medical image, and an extent of inspiration of lungs of theimaging subject.
 5. The method of claim 1, wherein the trained deepneural network, the image quality metric, and the image qualitycriterion, are indexed in non-transitory memory based on an imagingprotocol used to acquire the medical image.
 6. The method of claim 1,wherein the image quality metric is an angle of rotation of an imagingsubject with respect to a projection plane of the medical image, andwherein determining the image quality metric based on the positionalattribute of the anatomical feature includes: determining a firstdistance between a first location of a first anatomical feature and asecond location of a second anatomical feature; determining a seconddistance between the second location of the second anatomical featureand a third location of a third anatomical feature; determining a ratiobetween the first distance and the second distance; and determining theangle of rotation of the imaging subject with respect to the projectionplane based on the ratio.
 7. The method of claim 6, wherein determiningif the image quality metric satisfies the image quality criterioncomprises: accessing a pre-determined rotation range; comparing theangle of rotation with the pre-determined rotation range; and respondingto the angle of rotation not falling within the pre-determined rotationrange by updating a status of the image quality criterion to indicatethe angle of rotation does not match the pre-determined rotation range.8. The method of claim 1, wherein mapping the medical image to thepositional attribute of the anatomical feature using the trained deepneural network comprises; mapping the medical image to a segmentationmask of the anatomical feature; and determining one or more of a length,a width, a center point, an area, and an orientation of the anatomicalfeature, based on the segmentation mask.
 9. The method of claim 1,wherein the image quality metric is an extent of clipping of theanatomical feature, and wherein determining if the image quality metricsatisfies the image quality criterion comprises: determining if theextent of clipping of the anatomical feature is below a threshold extentof clipping; and responding to the extent of clipping of the anatomicalfeature being below the threshold extent of clipping by: setting astatus of the image quality criterion to a value indicating thecriterion is satisfied.
 10. A medical imaging system comprising; animaging device; a user input device; a display device; a memory, whereinthe memory includes machine readable instructions, and a trained deepneural network; and a processor, wherein the processor is communicablycoupled to the imaging device, the user input device, the displaydevice, and the memory, and when executing the machine readableinstructions the processor is configured to: receive an imaging protocolselection via the user input device, wherein the imaging protocolselection is indexed to the trained deep neural network and an imagequality criterion; acquire a medical image of an anatomical region of animaging subject with the imaging device based on the imaging protocolselection; map the medical image to an attribute of an anatomicalfeature using the trained deep neural network; determine a quantitativeimage quality metric based on the attribute of the anatomical feature;determine if the quantitative image quality metric satisfies the imagequality criterion; and display the medical image and the quantitativeimage quality metric, via the display device.
 11. The medical imagingsystem of claim 10, wherein the quantitative image quality metric is aclassification score, wherein the classification score indicates aprobability of the medical image including the anatomical feature, andwherein the image quality criterion comprises a classification scorethreshold.
 12. The medical imaging system of claim 11, wherein the imagequality criterion is satisfied by the classification score being greaterthan the classification score threshold, and wherein the image qualitycriterion is not satisfied by the classification score being less thanthe classification score threshold.
 13. The medical imaging system ofclaim 10, wherein, when executing the instructions, the processor isfurther configured to: respond to the quantitative image quality metricnot satisfying the image quality criterion by making a first adjustmentto the imaging device.
 14. The medical imaging system of claim 10,wherein, the processor is configured to display the medical image, thequantitative image quality metric, and the image quality criterion, viathe display device by: highlighting the anatomical feature; andsuperimposing the quantitative image quality metric over a region of themedical image proximal to the anatomical feature.
 15. The medicalimaging system of claim 10, wherein, when executing the instructions,the processor is further configured to: respond to receiving the imagingprotocol selection by pre-loading the trained deep neural network fromthe memory prior to acquiring the medical image.
 16. A methodcomprising: receiving an imaging protocol selection; acquiring a medicalimage based on the imaging protocol selection; mapping the medical imageto a first positional attribute of a first anatomical feature using atrained deep neural network; mapping the medical image to a secondpositional attribute of a second anatomical feature using a secondtrained deep neural network; determining an image quality metric basedon the first positional attribute of the first anatomical feature andthe second positional attribute of the second anatomical feature;determining if the image quality metric satisfies an image qualitycriterion; and responding to the image quality metric satisfying theimage quality criterion by: displaying the medical image, the imagequality metric, and an indication of the satisfied image qualitycriterion, via a display device.
 17. The method of claim 16, wherein thefirst trained deep neural network and the second trained deep neuralnetwork share one or more feature encoding layers.
 18. The method ofclaim 16, wherein the first positional attribute is a first segmentationmask of the first anatomical feature, and the second positionalattribute is a second segmentation mask of the second anatomicalfeature.
 19. The method of claim 18, wherein the image quality metric isan extent of overlap between the first anatomical feature and the secondanatomical feature, wherein the extent of overlap between the firstanatomical feature and the second anatomical feature is determined basedon an extent of intersection between the first segmentation mask and thesecond segmentation mask.
 20. The method of claim 19, whereindetermining if the image quality metric satisfies the image qualitycriterion comprises determining if the extent of overlap between thefirst anatomical feature and the second anatomical feature is less thana threshold extent of overlap.